Predicting social experience from dyadic interaction dynamics: the BallGame, a novel paradigm to study social engagement

Theories of embodied cognition suggest that a shared environment and ongoing sensorimotor interaction are central for interpersonal learning and engagement. To investigate the embodied, distributed and hence dynamically unfolding nature of social cognitive capacities, we present a novel laboratory-based coordination task: the BallGame. Our paradigm requires continuous sensing and acting between two players who jointly steer a virtual ball around obstacles towards as many targets as possible. By analysing highly resolved measures of movement coordination and gaming behaviour, game-concurrent experience ratings, semi-structured interviews, and personality questionnaires, we reveal contributions from different levels of observation on social experience. In particular, successful coordination (number of targets collected) and intermittent periods of high versus low movement coordination (variability of relation) emerged as prominent predictors of social experience. Importantly, having the same (but incomplete) view on the game environment strengthened interpersonal coordination, whereas complementary views enhanced engagement and tended to generate more complex interactive behaviour. Overall, we find evidence for a critical balance between similarity and synchrony on the one hand, and variability and difference on the other, for successful engagement in social interactions. Finally, following participant reports, we highlight how interpersonal experience emerges from specific histories of coordination that are closely related to the interaction context in both space and time.

For each trial, we selected nine out of 15 possible obstacles.Our semirandom procedure involved picking a number of obstacles from four pre-defined groups: (1) small obstacles right next to targets (green): two out of the three small inner circle obstacles were randomly selected, as well as the outer small obstacle on the direct line between the central and peripheral targets that was not yet covered; (2-4) areas between two outer targets (yellow, red and blue): in each area between two peripheral targets, two out of three obstacles were randomly selected.In the present example, a possible selection of obstacles is indicated by darker colour shades and continuous border lines.Note that the visibility of the nine selected obstacles (to each player) was selected at random, in line with the current play condition (SAME/individual play or DIFF).

Supplementary Materials B
Movement coordination measures 1. Detailed methods

Windowed-lagged cross-correlation:
In line with a recent validation study (Schoenherr et al, 2019), as well as our own tests with a range of parameter settings, we settled on windows of 3.6 seconds (220 frames) and a window overlap of 3.3 seconds (200 frames), leading to a step-size of 0.3 seconds.We further assessed correlation between players' movements at interpersonal lags of up to 1.1 seconds (70 frames) with single frame increments (60 Hz resolution, i.e. 0.017 seconds).
In other words, we assessed correlation across 141 lags (70 lags player one leads, 70 lags player two leads, 1 lag simultaneous movement) at 157 time points during each trial.In other words, we assume that events last approximately 3.6 seconds, that the evolution of players' interactions can be observed at a resolution of 0.3 seconds, and that inter-player coordination does not exceed a delay of 1.1 seconds and can be observed at a resolution of 0.017 seconds.

Peak-picking:
To identify potential leading or lagging between players at each of the 157 time steps, we identified the lag between players' steering directions that showed the strongest correlation above a minimum of 0.3 that was closest to simultaneous movement (0 lag).Since we set time steps at which the maximal correlation coefficient over all lags is lower than 0.3 to zero, our ppWLCC parameters indicate coordination over short, intermittent periods of time Mutual Information: MI is a framework that allows for the quantification of shared information between two signals and is based on Shannon's entropy from information theory.Entropy is computed by binning the given data set, and calculating the probability of a given data point to fall into either of these bins.These probability values are then multiplied with the logarithm of the probabilities, summed and multiplied by minus one to return to a positive scale.MI is calculated by adding the individual entropies of the two signals and then subtracting their joint entropy (Cohen, 2014).Using custom-made scripts based on Cohen (ibid), we calculated MI values for each block, that is, in parallel with moments of experience rating in the experiment.A block comprises three or four trials.Accordingly, we calculated MI for durations of 180 or 240 seconds, as well as for each session, condition and pair.The number of bins to discretise the data was estimated using the Freedman-Diaconis rule (Freedman & Diaconis, 1981).As the number of bins influences the entropy estimate, we first estimated the optimal bin number for each pair, session, condition, and block.Then, we took the ceiling of the grand average of the optimal bin number (25 in our data set) and re-ran the whole analysis with 25 bins for every calculation.In order to derive standard statistical Z-values of our MI estimates, we additionally applied permutation statistics.In 500 iterations, we temporally shifted one of the two time series by some random factor and calculated MI, to finally create a distribution of MI values expected under the null hypothesis from which Z-values were derived (again compare Cohen, 2014).These Z-values (one for every pair, session, condition, and block) were then subjected to statistical comparisons.

Phase-slope-index:
As for MI, we aggregated data at the level of the block.We estimated PSI (using data2psi.m,METH toolbox, Guido Nolte) for segment lengths ranging from 20 samples (0.333 seconds at a sampling rate of 60 Hz) to 240 samples (4 seconds) in steps of 20 samples (0.333 seconds) but for further analysis, we selected PSI values calculated using a time window of 40 samples (0.667 seconds).This choice was motivated by systematical assessment of the parameter space in an initial analysis step with the goal of finding the segment length that maximises the mean normalised PSI (calculation of the standard deviation across epochs using the jackknife method was done with different epoch lengths of 2, 3, and 4 seconds) across pairs, sessions, conditions, and blocks.What is more, PSI was calculated across all frequencies (wide band) based upon visual inspection of the grand average power spectra, where no apparent frequency peaks nor differences between conditions were present.

Thematic content analysis
Most of the themes identified in our thematic content analysis reflect participants' answers to one of our interview questions.For instance, we asked participants what it was like to play alone after the joint play period, and one of the themes that emerged from our analysis was 'Individual play'.Interestingly, we neither asked participants about the game environment, nor the technical setup, themes that nearly all players spoke about both in the early, openly phrased questions, as well as at later stages of the interview.Below, we summarise participant statements in each theme.
Game environment.The visual appearance of the game environment was present in detail to participants during the interview.This is observable from remarks such as "there was R2D2 on the coins..", or "the red bars come to mind..".Participants associated targets with a sense of reward, were eager to find a good path and avoid obstacles, including remembering the location of invisible obstacles.For instance, many players reportedly sought to "find the shortest / easiest path!", "go fast from A to B" and often recalled "the obstacles.. being slowed down" or, "especially when together -where are the invisible obstacles?".Participants also used the game environment to describe strategic considerations -"this moment, before each trial -we could think about which path to take"; or, "it's better go in curves, the direct path is usually blocked".Participants further mention their sense of (not) being able to reach coins, steer the ball faster, avoid obstacles or get off them quickly.For instance, many players expressed that "it was complicated -is there an invisible obstacle?Are we in disagreement?This not-knowing..".Another player pointed out that it was "frustrating to be stopped right before the coins, even more when playing together".On the other hand, many players described the pleasure of "that moment of -ah, here is a path!".
Positive emotion.Participants clearly expressed their overall positive experience of the game: "it was fun!I didn't check the time"; "really cool experiment!No 'experiment experience' for me!"; "I was deeply engaged"; "I was motivated to collect the coins -you really enter the game!".They also referred to the rewardingly challenging nature of the task -"it's not easy but you can figure it out!many small moments of success -difficult, but, we made it: I am steering by myself!that's how it felt!".Besides success ("reaching targets"; "when we managed to quickly identify our path"), positive emotions were was associated with joint play -"fun?playing together!not only coins, also coordinating!Learning about the other, how to coordinate"; "being able to steer through small paths between obstacles -nice, he thinks exactly like me!"; "when it worked well together, I had to suppress laughter!";"being fluent"; "on one wavelength"; "together and unanimous -nice!"; "over time, more control, new paths, more fluent"; or "after a while, having found a rhythm, coordinating just through our finger-movements, cool".Another group of positive statements was expressed around ease: "the steering was easier than I thought!"; "moving the ball, easy, like an extension of the body"; "when you had a feeling for what the other wants, then it was easy / good"; "the back and forth strategy -I was relieved to see that we easily settled on that".Another group of positive statements was made about individual play: "fun?playing alone.when together, there were irritations"; or, "alone much more fun, invisible obstacles easy.. yes..".
Negative emotion.Participants described sensations of exhaustion and frustration.They frequently mentioned "these annoying invisible obstacles", and complained ("come on…!") about coordination difficulties -"when the ball goes into another direction, not the one I wanted"; "when you tried, but you were stopped"; "obstacles, being blocked, slowed down, not being able to have my way -that was frustrating"; "when the other sees the obstacle, but cannot simply decide where we go -when it took longer than necessary to solve a block/obstacle".Confusion or lack of clarity was another source of negative mood: "somehow it was a bit stressful, not knowing what the other wants now"; "steering was somehow strange, as if it didn't always react the same way"; "exhausting.I felt like I couldn't do much -I tried to listen and respond to her strategysometimes it worked, sometimes it didn't".Participants further reported boredom and tiredness during the late part of the game: "exhausting?The duration / repetition"; "the coins appeared at predictable spots, that could be exploited (back and forth strategy / monotony)"; "even though I had mastered the finger-thing, it was more exhausting in the end, I simply couldn't focus, tired", or, "the control tasks were really exhausting, just staring at the screen".
Social presence.Particularly the early trial was described as social: "I focused, consciously, especially in the beginning of a trial: where does my partner want to go?"; or, "when playing together -waiting, what does my partner want -is what I want against this?once this is settled, it is like playing alone".In general, participants emphasised mutual consideration and described a process of attuning and coming to agreement: "more just coordinating, coming to agreement.."; "working as a team, trying to understand the intentions of the other"; and "coordination, thinking of one's partner, being considerate -brings us faster to our goal!".There were also remarks about flow and fluency: "I had a good feeling for my partner -she didn't block me and we could always find a path together"; "when it was clear what we wanted to do, then my partner was especially present to me".On the other hand, many participants report a sense of social presence in moments of disrupted flow: "thinking of my partner?When blocked..".Relatedly, a group of comments evolved around confusion and the indirect nature of communicating: "when it didn't go well -why?What does she want?Does she have an obstacle?Why can't we move on"; "in part funny -you don't know if it's an invisible obstacle or your partner who doesn't collaborate"; "not knowing when you see different things.."; and "not knowing what the other is doing..", hence "you make an effort, try to communicate, signal.." -"steering, collecting coins with someone you can't see, no verbal back and forth".For some, their partner felt present the whole time -"when it didn't work, or, when it went really well -increasingly, over the course of the game"; "I had a 'wefeeling', rather than thinking of her explicitly.."; "throughout the game -where does he go, what does he want, I want him to go here now, or, argh, he wants something elSF.. everything really!".
Strategy.Participants talked about strategies to coordinate as well as reach pragmatic goals.They described "waiting, to know where the partner wants to go, then support her"; "trying to perceive the other, their intentions -especially in the beginning of a trial".They also sought to keep track of "invisible obstacles -know where they are, so I can let my partner lead around them", and "notice when partner wants to avoid an obstacle -respond to that, because I cannot see it.Listen".To avoid risks ("it's difficult enough to coordinate with the other -better keep it simple!"),many preferred a 'back-and-forth' strategy: "find simplest path between two targets, and stay there".However, some participants wanted to explore, "not just go back and forth", "especially when alone -try out more".Finally, some remarks highlight the intuitive nature of finding a workable path: "steering the ball, like, so that it moves -if it doesn't, better 'let go', go with the other"; or, "this direction doesn't work?Stay calm, make a few moves, see where it wants to go".
Individual play.Many participants voiced a sense of ease ('relaxed') at the transition to individual play -it was easier to play the game -"to know where the invisible obstacles are, when playing alone, easy!"; "it was better to play alone -you could control the ball, find and stick to the shortest path".Some even described a more active sense of engagement: "when I played alone, I felt more active"; "I really know how this works now, the steering, and I can choose"; "I explored more when I played alone".Other participants reported a need to readjust: "the steering was unusual -my partner must have kept me on track before!"; "steering a really nice smooth curvethat was better together, being 'fluent'!";or even felt that the ball moved slower, "alone was slower than together!";"I felt like I was slower, alone" .Many participants further felt bored or 1 missed their partner: "the steering was ok but it was a bit boring.. together it was more fun!"; "just going through the motions, working it off"; "it was a bit boring in the end -I missed my partner, when I didn't want to move, she helped me"; "in the beginning of the session, it was fun to play alone (I didn't yet know how to play) -in the end not anymore, it was boring".Finally, several participants found joint and individual play more or less the same -"back and forth, if it doesn't work, swap route -very similar alone and together!";"not such a big difference, similar performance", or, "sort of evens out -easier because of the ball control, more difficult because of three completely invisible obstacles".
Technical comments.Nearly all participants commented on the laboratory environment, either speculating about the technical implementation or mentioning their bodily experience in relation to the setup.For instance, participants were "thinking about how you realised this -two separate rooms, steering one ball together, all the recordings.."; "noticed the symbols in the While acceleration was reduced when playing alone, the maximum speed of the ball was the same. 1 corners -for the eye-tracker?";found it "easy to understand -the game mechanism, with the landscapes etc"; or remarked on the "unusual game control!".They also mentioned their fingers (steering), their back (sitting) and eyes (staring at the screen).Participants also described general shifts in experience over time -"first figuring out, then fun, then repetitive (lack of focus)"; "at first it was difficult to know what she wants, that got better over time -okay, I have to let go sometimes, notice where she wants to go", as well as, "the steering got very easy, over time".2. Leave-one-out cross-validation of the final models' fixed effects

Differences
The cross-validation of fixed effects yielded highly significant and strong correlation between predicted and observed experience ratings, speaking to the generalisability of our findings -in the model of engagement, this is true for repeated measures (REP) (r = .454,p < .001),but not when aggregating over time (AVRG) (r = .12,p = .58),which is in line with the significant impact of the factor time on engagement.For the models of agreement and predictability, our cross validation yielded highly significant correlations between predicted and observed ratings, both for repeated measures as well as the pair average (agreement: (REP) r = .433,p < .001;(AVRG) r = .486,p = .019;predictability: (REP) r = .454,p < .001;(AVRG): r = .632,p = .001).
Note that the random intercept of the final models captured a greater part of the total variability in the model of engagement (67%), compared to agreement (54%) and predictability ratings (52%).Overall, generalisability hence appears higher for the latter two models.Relating absolute interpersonal time lag (ignoring who is the leader) against absolute agreeableness differences (ignoring who is more agreeable), we find no significant correlation (r = .12,p =.578).(B) Excluding the two outliers (pairs with agreeableness differences > 2SD above the mean), agreeableness difference and interpersonal time lag correlate significantly (r = .52,p = .016).(C) Consider directed values for both interpersonal time lag and agreeableness, we see no significant correlation.(D) When considering differences in performance (number of targets collected during individual play), we see that the better player tends to be the leader during joint play (r = -.61,p = .002).
3. Potential link between agreeableness differences and leader-follower dynamics (interpersonal time lag)  2. Main effects of trial segment (basic movement) 3. Additional interaction effects (trends) Figure SA.1 Possible obstacle locations and semirandom selection procedure.For each trial, we selected nine out of 15 possible obstacles.Our semirandom procedure involved picking a number of obstacles from four pre-defined groups: (1) small obstacles right next to targets (green): two out of the three small inner circle obstacles were randomly selected, as well as the outer small obstacle on the direct line between the central and peripheral targets that was not yet covered; (2-4) areas between two outer targets (yellow, red and blue): in each area between two peripheral targets, two out of three obstacles were randomly selected.In the present example, a possible selection of obstacles is indicated by darker colour shades and continuous border lines.Note that the visibility of the nine selected obstacles (to each player) was selected at random, in line with the current play condition (SAME/individual play or DIFF).
2. WLCC Surrogate statistics: real vs surrogate correlation across interpersonal lags Surrogate calculation: We calculated surrogate WLCC on data of players from distinct pairs moving through the same game landscape.To take into account the potential influence of both temporal and spatial context, we preserved the temporal structure of the trial as well as the spatial arrangement of targets and obstacles.Figure S.1 illustrates the mean (and ci of the mean) WLCC for real compared to surrogate pairs across the interpersonal lags we considered in the remainder of our analyses.Both statistics are based on mean values across all pairs and blocks of joint play.

Figure SB. 1
Figure SB.1 Synchrony in real and surrogate pairs.Mean windowed-lagged cross-correlation coefficients are displayed for real and surrogate pairs separately across interpersonal lags of up to 1.27 seconds.The shaded area indicates the confidence interval (ci) around the mean.

Figure SC. 1
Figure SC.1 Ball-velocity of players from weakly compared to strongly coordinated pairs, during joint and individual play of the second session.Both plots show histograms of momentary (frame to frame) ball velocity during joint (blue) compared to individual play (red).Note that maximum velocity is limited to 2.6 during individual, and 3.1 during joint play.The same number of data points is considered in all four cases: we considered the maximally and minimally coordinated 11 pairs (excluding the central pair), as well as 20 trials of joint play, versus 10 trials of individual play from two players.While weakly coordinated players move faster during individual play (left plot), ball velocity is similar during joint and individual play of strongly coordinated players.

Figure SD. 1
Figure SD.1 Results from the leave-one-out cross-validation of the fixed effects in the final models of participants' engagement, agreement and predictability ratings (left to right).The plots show repeated measures correlations of observed (real) ratings with those predicted based on data from all but the present pair (12 x 2 values per pair).The dashed bar and statistics in the bottom right of the plot indicate the group average repeated measures fit -solid lines present individual pair fits.

Figure SD. 2
Figure SD.2 Interpersonal time lag in finger movement coordination as a function of interpersonal difference in agreeableness -and performance?The plots show mean values of interpersonal time lag across all joint play trials, as well as the difference between players in the NEO-FFI agreeableness score (except Figure SD.2D, which includes performance differences instead of personality differences).(A)Relating absolute interpersonal time lag (ignoring who is the leader) against absolute agreeableness differences (ignoring who is more agreeable), we find no significant correlation (r = .12,p =.578).(B) Excluding the two outliers (pairs with agreeableness differences > 2SD above the mean), agreeableness difference and interpersonal time lag correlate significantly (r = .52,p = .016).(C) Consider directed values for both interpersonal time lag and agreeableness, we see no significant correlation.(D) When considering differences in performance (number of targets collected during individual play), we see that the better player tends to be the leader during joint play (r = -.61,p = .002).

Figure SF. 3
Figure SF.3 Significant Effects of Trial Segment in Basic Movement and Movement Coordination Measures.While participants tend to change finger movement direction less frequently in the final trial segment (Figure SF.3A), the movement amplitude (aggregate finger movement) does not decrease (Figure SF.3B).Overall, the results indicate that participants first increase their finger movement and then settle onto fewer, larger movements in the final trial segment.Within-class of observation FDR corrected p-values are indicated by *** (p < .001),** (p < .01),* (p < .05)and + (p < .10).
at the transition from joint to individual play

Table SE .
3 Post hoc ANOVAs -paired comparisons of individual blocks and trial segments for significant main effects of block and trial segment, respectively.p-valuesareFDRcorrected within each family of observation -together with the ANOVAs for the effects of session, condition and block, as well as posthoc paired comparisons for significant interaction effects.Significance levels are additionally indicated by *** (p < .001),**(p<.01),*(p<.05),+(p<.10).TableSE.4MANOVA and post hoc ANOVAs results for the effect of trial segment.p-values are FDR corrected at the MANOVA level (together with the MANOVAs of session, condition and block), as well as within each family at the ANOVA level (together with the ANOVAs of session, condition and block).Additionally, significance levels are indicated by *** (p < .001),** (p < .01).